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ABSTRACT 

This study was designed to develop a methodology for 
evaluating the application of Instructional Technology (IT)., 
conceived of in terms of a systems approach. . An idealized model of IT 
was prepared based on a literature review of current models. The 
model was matched against basic nursing education programs purporting 
to use IT. A rating technique was used to measure the "goodness of 
fit«»^ Strengths and weaknesses of each program are identified as well 
as general trends in IT use. Six of the most effective programs were 
selected for validating the methodology. Data supporting the approach 
is presented. (Author) 
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Introtluction 



Individualized instruction is a concept represented by a wide variety 
of processes and products. This is as it should be in a relatively yonng 
area in which new approaches are constantly being tested and revised. 
However, efforts to evaluate the effectiveness of the "technology" through 
which individualized instructional development is implemented in the real 
world suffers from this diversity. If a methodology could be devised which 
represented a "proven" (i.e. validated) approach to assessing the effective- 
ness of individualized program/product development, it could be used as a 
"universal" measure, cutting across specific approaches, media, products 
and learner populations. In short, if therp is a technology of instructional 
development, then a method for determining the degree of 'fit becv/een that 
technology and a particular instructional effort would represent one measure 
of its effectiveness. The development and validation of just such a method 
is the subject of this paper. 

The purpose of the project on which this research v/as based was to 
evaluate the application of instructional technology in basic nursing educa- 
tion. The study was to be national in scope and include nursing education 
programs from all geographical regions of the country. The study was also 
to include the different types of nursing education programs~4-year bacca- 
laureate,, 2-year associate degree, and hospital -based diploma programs--as 
well as soine which had received federal grants expressly for implementing 
or expanding instructional technology (IT). In addition, the study was to 
include nursing programs whose application of IT involved different subject 
matter areas and student learning patterns. Finally, IT was to be regarded 
as a systems process of instructional development rather than simply as the 
use of "technology" (media) in instruction. 

Several objectives were to be achieved by the evaluation. One v/as to 
assess the current overall state-of-the-art of IT in nursing education. A 
second was to identify those specific aspects of IT being applied effectively 
and those not. A third was to identify student learning patterns positively 
affected by well-conceived IT efforts. The last objective was to provide 
recommendations for improving tlie effectiveness of IT in nursing education. 



The focus of this paper is the methodology used to conduct the evalua- 
tion, flajor conclusions and recoimiendations of prime interest to the 
sponsoring agency generated from the study and detail about the constraints 
encountered in carrying out the study are not considered here. 

Methodology 

Defining IT 

The initial task was to clarify the concept of instructional technology. 
Most current descriptions of IT generally encompass the total process of 
developmental activities. For purposes of this effort it was necessary to 
establish in detail the elements of this process. A synthesized or "idealized" 
IT Model was constructed from an analysis of several extant models developed 
in education, training, and research. Models by Briggs (1970), Glaser (1965), 
Kaufman (1968), Mager (1957), and Tosti and Ball (1969) plus mv^dels developed 
by AIR in connection with its work on Industrial and military training 
projects v/ere analyzed to identify common components* The resu'its of this 
work formed the basis for the idealized IT Model shown on the following page. 

This synthesized IT Model consists of five major phases with each phase 
comprised of varying numbers of steps and substeps. It constituted our 
definition of v/hat instructional technology should be, given ideal conditions 
(i.e. adequate funding, appropriately trained personnel, time to accomplish 
all steps in an orderly fashion, etc.). The Model was, in effect, the baseline 
or the standard against which the specific and "real world" nursing educa- 
tion program's application of IT was compared. While it was not expected 
tliat any nursing program v/ould have performed all tie steps and substeps, 
the Model nonetheless was the constant for compari.iy actual applications 
of IT. 

Quantifying the Application of IT 

The second major task was to devise a method for quantifying nursing 
program performance in carrying out the IT process as defined. This was 
accomplished by identifying and weighting the various options by means of 
which certain substeps could be perfoniied. Since a fundamental premise of IT 
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is that all necessary decisions be based on empirically derived data and 
not subjective assumptions, our approach was to assign more weight to those 
options .vhich yield systematic, objective information and less or no 
weight to those based on soft data or opinions. 

The mechanics for establishing the weighting scheme consisted of listing 
the various methods/procedures by v/hich each of the pertinent substeps could 
be performed (52 of the 86 substeps of the Model could be so analyzed). 
The alternative methods/procedures generally ranged from not performing the 
substep at all through "doing it on the basis of subjective or traditional 
assumptions" to "completing it by empirical data-oriented processes." 

Four AIR staff members independently rated each method/procedure listed 
for each pertinent substep by the following criteria: 

• A rating of "3" was to be assigned if the method/procedure yields 
objective information, is quantitative in nature, and minimizes 
reliance upon subjective judgments. (Example: Assessing entry 
behavior by pretests and/or skill demonstrations.) 

• A rating of "2" was to be assi^^ed if the method/procedure yields 
data having both objective and subjective components but makes use 
of information and resources typically available to nurse educators 
and is reasonable in relation to "real world" constraints. (Example: 
Assessing entry behavior by examining previous grades and educational 
achievement.) 

. A rating of "1" was to be assigned if the method/procedure yields 
data based upon non-systematic and subjective input. (Example: 
Assessing entry behavior on the basis of year level.) 

• A rating of "0" was to be assigned if the substep was not performed. 



The determination of the final v/eight assigned to each method/ procedure 
was based upon concili tating the independent ratings of the four raters as 
follows: 

• When the method/procedure was given the same rating by three or more 
of the raters, it was assigned that weight (e.g. individual ratings 
of 3, 3, 1, 3 = a weight of 3). 

• When the method/procedure was given the same rating by two raters 
and the mean of all four raters was a whole number, the whole number 
was assigned as the weight (e.g. individual ratings of 2, 2, 1, 3 = 
a weight of 2) . 

^ When the method/procedure was given the same rating by two raters 
but the average of all four raters was not a whole number, the final 
weight was arrived at through negotiation (e.g. individual ratings 
of 2, 2, 1, 1 = 6/4 = 1.5. Final weight of 1 or 2 determined by 
consensus of raters). 

• When the method/procedure was not rated the same by any rater, the 
final weight was determined by negotiation and consensus. 

This scoring schema thus generated a means for ass' ning numerical 
scores to individual nursing education programs' effort in applying the 
IT process for developing instructional materials. 

Collecting and Scoring Data 

The next major task was to collect information about how IT was actually 
applied by nursing education programs. Interview schedules and mailed ques- 
tionnaires were developed around the IT Model. These data instruments (which 
were administered to a sample cf 33 nursing education programs selected, 
among other reasons, on the basis of location, types of program, and prior 
demonstrated instructional development experience) asked for information about 
whether, when, and how each step and substep of the IT Model was performed. 
Depending upon the answers given, each nursing program's performance could 
be assigned a numerical score. For example, nursing program "X" 
which perfon^iGd most of the substeps by emplrically-oriented methods/procedures 
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received a higher "IT score" than did nursing program "Y" which used more 
subjective methods/procedures or did not perform one or more of the substeps 
at all. 

The "TT scores" thus obtained were useful for a number of evaluative 
purposes. First, each nursing program's score could be compared against the 
total (ideal) score possible as a gross index of effectiveness. Second, 
each nursing program's score could be compared against other program's scores 
as a relative measure of effectiveness with other institutions independent of 
scope or content of the instruction developed. Third, a summation of scores 
across programs by IT phase, could be used to compare the relative effec- 
tiveness with v/hich each phase was performed and thus, identify those aspects 
of IT needing improvement. Fourth, an average of all "IT scores" across 
programs represented a general quantitative measure of the current state-of- 
the-IT-art in nursing education. 

Validating the IT Mode l 

The fourth major task v/as to demonstrate empirically the general iza- 
bility of IT for different learning environments, student populations, and 
learning patterns. The rationale underlying this task v/as that only if IT 
could be shown to be effective in facilitating learning in different opera- 
tional settings could there be any empirical basis of support for the IT 
Model. Six nursing programs v/hich received high "IT scores" (meaning that 
they developed instructional materials in close conformance to the IT 
Model) were consequently selected for a validation study. The objectives 
and criterion tests were reviewed by AIR project staff to verify that they 
accurately reflected the instructional materials developed by each of the 
respective nursing programs. Test data indicated achievement to criterion 
levels in all cases, providing support for the general validity and effec- 
tiveness of the IT Model. 
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Conclusion 



The iiiethodology described in this paper represents an effort to obtain 
evaluative information about an instructional development process consisting 
of many elements which can be performed by alternative methods. The method- 
ology can be applied regardless of the content area, the learning task, or 
the target population of the instructional development effort. The methodology 
generates a quantitative basis for comparing the quality of individualized 
instructional development efforts against an "ideal" standard and against 
the efforts of other similar organizations using IT. The method permits 
evaluating either system-v/ide instructional development efforts or indi- 
vidual efforts at one institution. 

The methodology described above can be improved in several ways. 
The analysis of extant IT Models could be broadened and larger numbers of 
judges used in selecting the critical common steps and substeps. Similarly, 
the identification of alternative methods/procedures for performing perti- 
nent substeps could be further refined. The rating and assignment of weights 
for the alternative methods/procedures could be more adequately accomplished 
through interrc .dr reliability coefficients. Finally, an evaluation research 
paradigm v/ould provide a more compelling scientific basis for evaluating 
the effects of IT than the validation effort reported. However, the current 
"version" of the methodology proved to be very useful in identifying areas 
of weakness in nursing applications of IT and in generating recommendations 
aimed at strengthening their efforts in these areas. 
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